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Abstract. This study assesses the impact of computer-based formative evaluations 
in an undergraduate English course for second semester kinesiology students 
at the Universidad Austral de Chile - Valdivia (UACh). The target of the course 
is to improve the students’ online reading comprehension skills in their field. 
A preliminary study was carried out in order to select the platform to implement 
the formative evaluations. Two platforms were available: a Dokeos-based system 
(Dokeos, 2015), and Questionmark Perception (Questionmark-Corporation, 2015). 
We performed a technical review of both platforms and an empirical test in a pilot 
group where students’ preferences were analyzed. Perception was selected since it 
proved to be the best choice in terms of functionality and ease of use. The target 
kinesiology group was divided into two subgroups, Gl, and G2, of the same size 
and similar composition in terms of gender. Each group had the opportunity to use 
the computer-based formative evaluations for one half of the semester, while the 
other group took traditional classroom lessons in the corresponding period of time. 
Most students using the computer-based approach had better performance than their 
counterparts in the actual exam, which was taken in a paper and pencil format. 
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1. Introduction 

English is the language of choice for business, science, and technology, which 
requires professionals in these disciplines to develop their English reading 
comprehension skills. We focus on English online reading comprehension, 
defined as the ability to read resources on a computer that offers capabilities like 
hypertext, access to multimedia, and web navigation. These capabilities constitute 
a fundamentally different medium than printed texts, offering a much richer 
experience to the reader, but demanding better skills to fully profit from this new 
media (Henry, 2005). The increase in books and texts available on the Internet 
has forced students to read more documents online than on paper, for which it is 
fundamental to develop systems to improve the development and evaluation of 
online reading comprehension skills. 

The Faculty of Medicine (FAME) at UACh receives very heterogeneous student 
groups in terms of their English proficiency levels. Programs of study at FAME 
require students to acquire competencies to communicate in their professional 
area. Furthermore, the need to get accreditation for linguistic communication 
competencies according to international quality standards makes it necessary for 
students to get used to the Computer-Based Assessments (CBA) systems used 
in international foreign language proficiency or English for Specific Purposes 
(ESP) tests. 

Both formative and summative evaluations at FAME are done in paper and 
pencil format, which involves a most time-consuming grading job for professors, 
reducing their available time for materials preparation and diversification of 
teaching strategies, also causing delays in the feedback delivery time. Motivated 
by these arguments, our study’s objective is to determine whether the introduction 
of formative CBA evaluations in ESP courses have in fact a positive impact on 
student learning at FAME. 

2. Method 

2.1. Preliminary study 

Objective: to select the software tool to use in our main study. 

Study group: 78 graduate students from a technical English course offered to the 
different Master and PhD programs at UACh were used for the initial survey. 
12 volunteers from that group participated in the empirical study. 
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Candidate platforms to implement formative evaluations: 

• SIVEDUC (Sistema Virtual de Educacion) based on Dokeos (2015). An 
evaluation development tool integrated to a Learning Management System 
(LMS) currently available at UACh; 

• Questionmark Perception, considered for several years as one of the most 
complete authoring systems, which is also integrated to an LMS (Hogan & 
Smith, 2005; Shulman, 2005). 


Table 1 . Initial survey results 


Characteristic of Students in the pilot group 

% 

Students owning a mobile computing device 

95 

Students with previous online computer based evaluation experience 

23 

Students that prefer taking online tests instead of paper and pencil 

62 

Students with previous experience that prefer taking online tests 

82 


Table 2. Technical evaluation comparison 


Questionmark’ s Perception vs SIVEDUC (dokeos based) 
Comparison Summary 


Perception 

SIVEDUC 

# of Question Types 

22 

8 

Question Type 

Developer's Preference 

End User’s Preference 

Essay 

Equal 

Equal 

Multiple Choice 

Equal 

Equal 

Ranking 

Perception 

Perception 

Column Match 

Perception 

Equal 

Cloze 

SIVEDUC 

Equal 

True or False 

Perception 

Perception 


An initial survey yielded students’ demographics and their attitudes and knowledge 
regarding the use of computer-based evaluations. This information was presented 
in Lazzeri, Gallardo, Cabezas & Leiva (2013). Table 1 summarizes these results, 
which let us conclude that the group had appropriate skills and motivation to take 
online formative assessments, and thus help us decide the tool to use in the main 
study. Afterwards, the group completed different exercises using SIVEDUC, 
and Perception. A final survey showed that over 70% of the students preferred 
Perception to answer each of the question types used: essay, multiple choice, cloze, 
ranking, column match, and True/Lalse. 
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We also performed a technical evaluation of both platforms, comparing primarily 
the number of types of questions offered, and the ease of use, both from the 
student/user and the professor/developer’s perspectives in terms of the number of 
actions required to carry out each task. This comparison also favored Perception, 
since it provided more types of questions, and, in most cases, it required less 
work for the developer to create and for the user to interact with the types of 
questions available in both systems. Table 2 summarizes these results, which led 
us to use Perception. 

2.2. Main study 

Figure 1. Sample formative evaluation: column match exercise and background 
material 5 


Question mart 




Reading 1: http://en.wikipedia.org/wiki/Kinesiology 


Questionmark 




Exercise 2- Match the following vocabulary. 

Drag the options on the right to match the choices on the left 


^BIOMECHANICS ^y 

^MUSCLES 

) 


^PHYSIOLOGY “n 

^NERVOUS SYSTEM ^ 


^ NEUROSCIENCE ^NEURONS 

^ ^BEHAVIOR 

) 


^ ANATOMY 




PERFORMANCE ^y 




^ ERGONOMY ^SPECIAL SEAT 

^ ^FLEXIBILITY 

) 


^ PSYCHOLOGY ^ FREUD 




Submit | 



V 

< 


> 



1 1 L 

Help 



@ http://en.wikipedia.org/ - Kinesiology - Wikipedia, the free encyclopedia - Intern... - n 

Create account Log in ^ 


•V 


Article Talk Read Edit View history [Search 

Kinesiology 


WikipediA 

The Free Encyclopedia From Wikipedia, the free encyclopedia 


Featured content 

Random article 
Donate to Wikipedia 


About Wikipedia 
Community portal 
Recent changes 
Contact page 
Tools 

What links here 
Related changes 
Upload file 
Special pages 
Permanent link 
Page information 
Wikidata item 
Cite this page 
Print/export 
Create a book 
Download as PDF 
Printable version 
Languages { 

Azerbaycanca 


Cestina 

Cymraeg 

Deutsch 

EMnviKci 

Espanol 


This article is about the scientific study of human movement. For th 
alternative medicine technique, see Applied kinesiology. 

Kinesiology, also known as 
human kinetics, is the 
scientific study of human 
movement. Kinesiology 
addresses physiological, 
mechanical, and 
psychological mechanisms. 

Applications of kinesiology 
to human health include 
biomechanics and 
orthopedics; strength and 
conditioning; sport 
psychology; methods of 
rehabilitation, such as 
physical and occupational 
therapy; and sport and 

exercise. 111 Individuals who a series of images that represent 
have earned degrees in research (left) and practice (right) in the field 

... , . of kinesiology 

kinesiology can work in 

research, the fitness 

industry, clinical settings, and in industrial environments. 121 Studies of 
human and animal motion include measures from motion tracking 
systems, electrophysiology of muscle and brain activity, various methods 
for monitoring physiological function, and other behavioral and cognitive 
research techniques. 131141 

Kinesiology as described above should not be confused with applied 
kinesiology, a controversial™ 71 medical diagnostic method. 

The word comes from the Greek idvnoi? kinesis, "movement" (itself from 
kiveTv Kinein, "to move"), and -AoYi'a -logia, "study." 



The computer-based formative evaluations were developed using Perception. The 
course’s instructor provided the contents of the evaluations in a word processor 
format, then the software administrator implemented them in Perception. This 

5. Screenshot reproduced with kind permission from Questionmark™ 
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implementation was very time consuming since the ability of Perception to import 
word processor documents or plain text files is very limited. Occasionally, it proved 
useful to employ a third party software, Respondus (2015), to simplify this task, 
but there was still a lot of work to be done manually. 

Formative evaluations consisted of one or more reading sections along with a set 
of exercises. Figure 1 shows a sample exercise from the first formative evaluation 
implemented in Questionmark. Table 3 presents the composition of the second 
formative evaluation used in the study. 

Table 3. Composition of second formative evaluation 


EXERCISE TYPE 

DESCRIPTION 

Technical Kinesiology Paper (pdf) 

4 pages, 3047 words 

Classification 

25 items 

Translation 

20 items 

Multiple Choice (4) 

20 items 

Cloze 

10 items 

Essay 

10 items 

Class Time 

90 minute 

Off Class Access 

Open, Unlimited 

Feedback 

Correct answers to each question 
answered were given every time responses 
were submitted 

Submissions allowed 

Unlimited 


A pilot study to determine the usefulness of the formative evalutions was conducted 
on the target kinesiology group, which was randomly divided into two subgroups, 
Gl, and G2, of the same size and similar composition in terms of gender. Each 
group had the opportunity to use the CBA Formative Evaluations for one half 
of the semester, while the other group took traditional classroom lessons in the 
corresponding period of time. We compared the performance of the subgroups in 
the actual exams which took place at the end of each phase. The actual exam was 
graded in a 1-7 scale, with a minimum passing grade of 4. 

3. Discussion 

During the first half of the semester (Phase 1), Gl, using CBA, obtained better 
grades than G2, which used a traditional approach to prepare for the corresponding 
exam. In the second half of the semester (Phase 2), the roles were reversed, 
and so were the results: G2, using CBA got the better results. In the intragroup 
comparison, both groups fared better when using CBA. At the individual level, 
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most students got better results using the CBA than attending the traditional 
lectures. The descriptive statistics in Table 4 show a clear trend favoring the use of 
CBA formative evaluations. A post-test survey showed that most students liked the 
CBA approach and found it useful and easy to use. 


Table 4. Summary of results obtained by students in both groups 




PHASE 1 

PHASE 2 

Effect of Technology on Performance 

Group 




Better 


Worse 

Equal 


n 

Mean Sdev 

n Mean Sdev 

n 

% 

n 

% n 

°/o 


G1 

28 

5.6 0.97 

28 5.29 0.739 

19 

68 

8 

28.5 1 

3.5 

G2 

29 

5.51 0.76 

27 5.76 0.715 

13 

48 

11 

41 3 

11 

Using CBA 


4. Conclusions 

We can conclude that the CBA formative evaluations had a positive effect on 
learning in this context, since all intergroup and intragroup comparisons favored 
students using CBA, and the surveys showed that students enjoyed the CBA 
approach and found it useful. Individual performances show different kinds of 
learners: most students did better with the CBA approach, but a few got better 
scores with the traditional approach. A more in-depth statistical analysis of the 
results and surveys can be found in Lazzeri et al. (2015). Our results are only 
directly applicable to our particular context. 
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